{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fe6e643-9453-4381-9445-bd471685fb96",
   "metadata": {},
   "source": [
    "# Labeling the [craigslist](https://huggingface.co/datasets/craigslist_bargains) dataset using Autolabel\n",
    "\n",
    "This is a multi-class classification task where the input are conversations between buyers and sellers and we have to correctly classify the item being sold into one of 6 categories. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "aacac4ae-c7f9-4dee-be3a-a2a6bfa099fa",
   "metadata": {},
   "source": [
    "## Install Autolabel\n",
    "Plus, setup your OpenAI API key, since we'll be using `gpt-3.5-turbo` as our LLM for labeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3dc19059-2f63-44b7-9a32-8b38a11249aa",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install 'refuel-autolabel[openai]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fbdeca2f-dd20-4634-b3f8-1b3ed45c4705",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# provide your own OpenAI API key here\n",
    "os.environ['OPENAI_API_KEY'] = 'sk-'\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8eb09918-494a-42ef-86a7-df93b3ce4284",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Download the dataset\n",
    "\n",
    "This dataset is available to install via Autolabel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b4c116ce-3294-45c2-9158-eac929f03a95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading example dataset from https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/craigslist/seed.csv to seed.csv...\n",
      "Downloading example dataset from https://autolabel-benchmarking.s3.us-west-2.amazonaws.com/craigslist/test.csv to test.csv...\n",
      "100% [........................................] [734774/734774] bytes\r"
     ]
    }
   ],
   "source": [
    "from autolabel import get_data\n",
    "\n",
    "get_data('craigslist')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5f038fcc-9ec0-4f2d-b84d-2e963656c6bd",
   "metadata": {},
   "source": [
    "This downloads two datasets:\n",
    "* `test.csv`: This is the larger dataset we are trying to label using LLMs\n",
    "* `seed.csv`: This is a small dataset where we already have human-provided labels"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "84b014d1-f45c-4479-9acc-0d20870b1786",
   "metadata": {},
   "source": [
    "## Start the labeling process!\n",
    "\n",
    "Labeling with Autolabel is a 3-step process:\n",
    "* First, we specify a labeling configuration (see `config.json` below)\n",
    "* Next, we do a dry-run on our dataset using the LLM specified in `config.json` by running `agent.plan`\n",
    "* Finally, we run the labeling with `agent.run`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "33d47b67-0718-4289-bb59-989d851d09ed",
   "metadata": {},
   "source": [
    "### First labeling run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c093fe91-3508-4140-8bd6-217034e3cce6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from autolabel import LabelingAgent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c93fae0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the config\n",
    "with open('config_craigslist.json', 'r') as f:\n",
    "     config = json.load(f)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1fad4e85-d598-413d-9b01-9690653a05ad",
   "metadata": {},
   "source": [
    "Let's review the configuration file below. You'll notice the following useful keys:\n",
    "* `task_type`: `classification` (since it's a classification task)\n",
    "* `model`: `{'provider': 'openai', 'name': 'gpt-3.5-turbo'}` (use a specific OpenAI model)\n",
    "* `prompt.task_guidelines`: `'You are an expert at understanding bank customers support complaints and queries...` (how we describe the task to the LLM)\n",
    "* `prompt.labels`: `['age_limit', 'apple_pay_or_google_pay', 'atm_support', ...]` (the full list of labels to choose from)\n",
    "* `prompt.few_shot_num`: 10 (how many labeled examples to provide to the LLM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6a3610fd-721e-44de-9b2c-2cd73ec86bba",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'task_name': 'CraigslistConversationClassification',\n",
       " 'task_type': 'classification',\n",
       " 'dataset': {'label_column': 'label', 'delimiter': ','},\n",
       " 'model': {'provider': 'openai', 'name': 'gpt-3.5-turbo'},\n",
       " 'prompt': {'task_guidelines': 'You are an expert at understanding conversations.\\n Given a text passage as input comprising of dialogue of negotiations between a seller and a buyer about the sale of an item, your task is to classify the item being sold into one of the following categories:\\n{labels}',\n",
       "  'output_guidelines': 'You will answer with just the the correct output label and nothing else.',\n",
       "  'labels': ['housing', 'furniture', 'bike', 'phone', 'car', 'electronics'],\n",
       "  'few_shot_examples': 'seed.csv',\n",
       "  'few_shot_selection': 'semantic_similarity',\n",
       "  'few_shot_num': 10,\n",
       "  'example_template': 'Input: {example}\\nOutput: {label}'}}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "acb4a3de-fa84-4b94-b17a-7a6fac892a1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an agent for labeling\n",
    "agent = LabelingAgent(config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "92667a39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8eea441e81724ed4998395cda5dd942b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┌──────────────────────────┬─────────┐\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Total Estimated Cost     </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $5.6125 </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Number of Examples       </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> 1210    </span>│\n",
       "│<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> Average cost per example </span>│<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\"> $0.0046 </span>│\n",
       "└──────────────────────────┴─────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┌──────────────────────────┬─────────┐\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mTotal Estimated Cost    \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$5.6125\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mNumber of Examples      \u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m1210   \u001b[0m\u001b[1;32m \u001b[0m│\n",
       "│\u001b[1;35m \u001b[0m\u001b[1;35mAverage cost per example\u001b[0m\u001b[1;35m \u001b[0m│\u001b[1;32m \u001b[0m\u001b[1;32m$0.0046\u001b[0m\u001b[1;32m \u001b[0m│\n",
       "└──────────────────────────┴─────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────── </span>Prompt Example<span style=\"color: #00ff00; text-decoration-color: #00ff00\"> ──────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────── \u001b[0mPrompt Example\u001b[92m ──────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">You are an expert at understanding conversations.\n",
       " Given a text passage as input comprising of dialogue of negotiations between a seller and a buyer about the sale \n",
       "of an item, your task is to classify the item being sold into one of the following categories:\n",
       "housing\n",
       "furniture\n",
       "bike\n",
       "phone\n",
       "car\n",
       "electronics\n",
       "\n",
       "You will answer with just the the correct output label and nothing else.\n",
       "\n",
       "Some examples with their output answers are provided below:\n",
       "\n",
       "Input: Buyer: Hi, I saw an add for this <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">6</span> bedroom apartment and I am interested in it. Seller: Helo, thank you for \n",
       "the interest. Let me know if you have any questions Buyer: I would like to offer $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">7696</span> for it Seller: I'm sorry \n",
       "that is <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">30</span>% lower than the listing price Buyer: Would you take $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9000</span>? Seller: I'd be willing to offer about <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10</span>% \n",
       "off.. at $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">10000</span> per month Seller:  Buyer: Can you do <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span> % off? Seller: $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9800</span> final offer Buyer: Ok, I can do that. \n",
       "Seller: great, thank you Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hi. I see you're checking out our beautiful apartment homes Buyer: Yes, they are nice but a bit \n",
       "expensive. Could you come down on the price? I would like to pay $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2000</span>. Seller: That price is much too low. These \n",
       "do come with a load of amenities. We have modern unites with free internet access and brand new appliances. Buyer: \n",
       "If I sign a <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">13</span> month lease, could you take $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2500</span>? Seller: Yes! That would be great Buyer:  Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hello! Are you interested in my apartment? Buyer: I work in NYC and had been renting there as well \n",
       "but its ridiculously expensive.   Is this per month?     I am just thinking close to <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1000</span>/month as my rent in nyc \n",
       "is <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1200</span> month Seller: Yes this is the per month price. I would not be able to go that low. This apartment is brand \n",
       "new, partially furnished, and has wood flooring. Similar apartments in the area are going for around $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1700</span> so this \n",
       "is really a steal! Buyer: how about <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1400</span>/month but I will put <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1000</span> down right now.  Seller: $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1450</span> is the lowest I \n",
       "can go Buyer:  Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: sure Seller: Hello, I would like to offer this for $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2550</span>.   Buyer: Sorry, I already sold it for <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2800</span> \n",
       "and this is messed up Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi, I am interested in the apartment.  Is there a lease?  Is it located in a good neighborhood? \n",
       "Seller: yes the community is very good and is very safe as it is gated. It has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> pools and a <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">24</span> hour fitness center\n",
       "for its residents Buyer: Great!  I see you want $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1487</span>, is there any way you can go down on the price a bit? Seller:\n",
       "The price is negotiable within reason Buyer: My offer is $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1040</span>. Seller: Thats a little low. Is it possible to meet \n",
       "more in the middle at $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1250</span>? Buyer: $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1200</span> is the most I can do. Seller: That sounds good to me Buyer: Great. Buyer:\n",
       "Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi is the apartment still available? Seller: Hi there! Yes, the apartment is still available and for \n",
       "sale, are you interested? Buyer: Yes I am interested does the price include utilities? Seller: Yes it does, but it \n",
       "doesn't include parking. We can negotiate a fair price and I can include parking as well if you're offer is decent.\n",
       "Buyer: Would you be willing to lease it for $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">400</span>? Seller: Would you be willing to bring it up to $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span>? Buyer: I\n",
       "can do $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>,<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">500</span> if you include the parking fees. Seller: I think that's acceptable for both of us, we can go ahead \n",
       "and submit this offer if that's alright with you? Buyer: That sounds good to me. Seller:  Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hi! Buyer: Hi there. Is the apt still available? Seller: yes it is. Full kitchen, beautiful inside \n",
       "out. No smoking an d no pet please Buyer: that works for me. How about the utilities? Seller: If you are willing to\n",
       "pay full price, I'll pay the utilities Buyer: I am willing to pay $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1165</span>.  Seller: How about $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1200</span>?  Buyer: Yes that\n",
       "works for me!  Seller: Great! just for your information, it will be <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> year lease and <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> month deposit based on your \n",
       "credit check and no eviction history Seller:  Buyer: sounds good! Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: hello, what is your offer? Buyer: Geesh, man, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2400</span> is a lot of money every month<span style=\"color: #808000; text-decoration-color: #808000\">...</span> how many \n",
       "bedrooms is this?  Seller: This is a <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span> bedroom and very spacious so it's worth the money. Buyer: Oh ok. That makes \n",
       "more sense. Seems like you have a studio also. Tell you what, I could do the three bedroom for <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2000</span>, or two bedroom\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1800</span>. I'll sign a <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> year lease either way  Seller: I can let you get the <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span> bedroom for <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2100</span> but that is as low as I\n",
       "can go. Buyer: Yeah, that's a big out of my range for me. You could probably do the <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span> bedroom for <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1800</span>, though?  \n",
       "Seller: That works, thanks. Seller:  Buyer: alright, sounds good Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi there. Is this available?  Seller: Yes sir it is Buyer: is it new or used?  Seller: It's used sir.\n",
       "I'm selling it for <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4400</span> Buyer: The price is way too high for me. Is there any room for negotiating here?  Seller: \n",
       "Sure. How much are you willing to pay?  Buyer: $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2200</span> please Seller: I won't be able to go that low..that's below \n",
       "the book value. What about <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4000</span>? Buyer: I can't afford more than <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3500</span>. \n",
       "Output: car\n",
       "\n",
       "Input: Seller: hello! Buyer: Hi!  Seller: Are you interested in my apartment listing Buyer: I am! Unfortunately I'm\n",
       "more in the budget for a space priced around $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">647</span> Seller: Have I mentioned that this amazing apartment comes with \n",
       "parking, and is within walking distance to BART? The view isn't much and it does have a lot of Sec <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">8</span> people but is \n",
       "fairly priced for it's size and location. I'll bottom line it for ya. I need the lease and am willing to cover your\n",
       "moving expenses and knock $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">300</span> off the monthly lease price. Buyer: That sounds like a great deal. I think I could \n",
       "swing $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">995</span> provided I get the part in an upcoming live animation simpsons film I've auditioned for.  Seller: I \n",
       "actually know the director! I'll put in a good word for you. Seller:  Buyer: \n",
       "Output: housing\n",
       "\n",
       "Now I want you to label the following example:\n",
       "Input: Buyer:  Buyer: hi. price is quite high but i like the apt.  Seller: Your offer is too low, I can not accept \n",
       "it. I need at least $<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2600</span> Seller: \n",
       "Output: \n",
       "</pre>\n"
      ],
      "text/plain": [
       "You are an expert at understanding conversations.\n",
       " Given a text passage as input comprising of dialogue of negotiations between a seller and a buyer about the sale \n",
       "of an item, your task is to classify the item being sold into one of the following categories:\n",
       "housing\n",
       "furniture\n",
       "bike\n",
       "phone\n",
       "car\n",
       "electronics\n",
       "\n",
       "You will answer with just the the correct output label and nothing else.\n",
       "\n",
       "Some examples with their output answers are provided below:\n",
       "\n",
       "Input: Buyer: Hi, I saw an add for this \u001b[1;36m6\u001b[0m bedroom apartment and I am interested in it. Seller: Helo, thank you for \n",
       "the interest. Let me know if you have any questions Buyer: I would like to offer $\u001b[1;36m7696\u001b[0m for it Seller: I'm sorry \n",
       "that is \u001b[1;36m30\u001b[0m% lower than the listing price Buyer: Would you take $\u001b[1;36m9000\u001b[0m? Seller: I'd be willing to offer about \u001b[1;36m10\u001b[0m% \n",
       "off.. at $\u001b[1;36m10000\u001b[0m per month Seller:  Buyer: Can you do \u001b[1;36m15\u001b[0m % off? Seller: $\u001b[1;36m9800\u001b[0m final offer Buyer: Ok, I can do that. \n",
       "Seller: great, thank you Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hi. I see you're checking out our beautiful apartment homes Buyer: Yes, they are nice but a bit \n",
       "expensive. Could you come down on the price? I would like to pay $\u001b[1;36m2000\u001b[0m. Seller: That price is much too low. These \n",
       "do come with a load of amenities. We have modern unites with free internet access and brand new appliances. Buyer: \n",
       "If I sign a \u001b[1;36m13\u001b[0m month lease, could you take $\u001b[1;36m2500\u001b[0m? Seller: Yes! That would be great Buyer:  Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hello! Are you interested in my apartment? Buyer: I work in NYC and had been renting there as well \n",
       "but its ridiculously expensive.   Is this per month?     I am just thinking close to \u001b[1;36m1000\u001b[0m/month as my rent in nyc \n",
       "is \u001b[1;36m1200\u001b[0m month Seller: Yes this is the per month price. I would not be able to go that low. This apartment is brand \n",
       "new, partially furnished, and has wood flooring. Similar apartments in the area are going for around $\u001b[1;36m1700\u001b[0m so this \n",
       "is really a steal! Buyer: how about \u001b[1;36m1400\u001b[0m/month but I will put \u001b[1;36m1000\u001b[0m down right now.  Seller: $\u001b[1;36m1450\u001b[0m is the lowest I \n",
       "can go Buyer:  Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: sure Seller: Hello, I would like to offer this for $\u001b[1;36m2550\u001b[0m.   Buyer: Sorry, I already sold it for \u001b[1;36m2800\u001b[0m \n",
       "and this is messed up Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi, I am interested in the apartment.  Is there a lease?  Is it located in a good neighborhood? \n",
       "Seller: yes the community is very good and is very safe as it is gated. It has \u001b[1;36m2\u001b[0m pools and a \u001b[1;36m24\u001b[0m hour fitness center\n",
       "for its residents Buyer: Great!  I see you want $\u001b[1;36m1487\u001b[0m, is there any way you can go down on the price a bit? Seller:\n",
       "The price is negotiable within reason Buyer: My offer is $\u001b[1;36m1040\u001b[0m. Seller: Thats a little low. Is it possible to meet \n",
       "more in the middle at $\u001b[1;36m1250\u001b[0m? Buyer: $\u001b[1;36m1200\u001b[0m is the most I can do. Seller: That sounds good to me Buyer: Great. Buyer:\n",
       "Seller: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi is the apartment still available? Seller: Hi there! Yes, the apartment is still available and for \n",
       "sale, are you interested? Buyer: Yes I am interested does the price include utilities? Seller: Yes it does, but it \n",
       "doesn't include parking. We can negotiate a fair price and I can include parking as well if you're offer is decent.\n",
       "Buyer: Would you be willing to lease it for $\u001b[1;36m1\u001b[0m,\u001b[1;36m400\u001b[0m? Seller: Would you be willing to bring it up to $\u001b[1;36m1\u001b[0m,\u001b[1;36m500\u001b[0m? Buyer: I\n",
       "can do $\u001b[1;36m1\u001b[0m,\u001b[1;36m500\u001b[0m if you include the parking fees. Seller: I think that's acceptable for both of us, we can go ahead \n",
       "and submit this offer if that's alright with you? Buyer: That sounds good to me. Seller:  Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: Hi! Buyer: Hi there. Is the apt still available? Seller: yes it is. Full kitchen, beautiful inside \n",
       "out. No smoking an d no pet please Buyer: that works for me. How about the utilities? Seller: If you are willing to\n",
       "pay full price, I'll pay the utilities Buyer: I am willing to pay $\u001b[1;36m1165\u001b[0m.  Seller: How about $\u001b[1;36m1200\u001b[0m?  Buyer: Yes that\n",
       "works for me!  Seller: Great! just for your information, it will be \u001b[1;36m1\u001b[0m year lease and \u001b[1;36m1\u001b[0m month deposit based on your \n",
       "credit check and no eviction history Seller:  Buyer: sounds good! Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Seller: hello, what is your offer? Buyer: Geesh, man, \u001b[1;36m2400\u001b[0m is a lot of money every month\u001b[33m...\u001b[0m how many \n",
       "bedrooms is this?  Seller: This is a \u001b[1;36m3\u001b[0m bedroom and very spacious so it's worth the money. Buyer: Oh ok. That makes \n",
       "more sense. Seems like you have a studio also. Tell you what, I could do the three bedroom for \u001b[1;36m2000\u001b[0m, or two bedroom\n",
       "\u001b[1;36m1800\u001b[0m. I'll sign a \u001b[1;36m1\u001b[0m year lease either way  Seller: I can let you get the \u001b[1;36m3\u001b[0m bedroom for \u001b[1;36m2100\u001b[0m but that is as low as I\n",
       "can go. Buyer: Yeah, that's a big out of my range for me. You could probably do the \u001b[1;36m2\u001b[0m bedroom for \u001b[1;36m1800\u001b[0m, though?  \n",
       "Seller: That works, thanks. Seller:  Buyer: alright, sounds good Buyer: \n",
       "Output: housing\n",
       "\n",
       "Input: Buyer: Hi there. Is this available?  Seller: Yes sir it is Buyer: is it new or used?  Seller: It's used sir.\n",
       "I'm selling it for \u001b[1;36m4400\u001b[0m Buyer: The price is way too high for me. Is there any room for negotiating here?  Seller: \n",
       "Sure. How much are you willing to pay?  Buyer: $\u001b[1;36m2200\u001b[0m please Seller: I won't be able to go that low..that's below \n",
       "the book value. What about \u001b[1;36m4000\u001b[0m? Buyer: I can't afford more than \u001b[1;36m3500\u001b[0m. \n",
       "Output: car\n",
       "\n",
       "Input: Seller: hello! Buyer: Hi!  Seller: Are you interested in my apartment listing Buyer: I am! Unfortunately I'm\n",
       "more in the budget for a space priced around $\u001b[1;36m647\u001b[0m Seller: Have I mentioned that this amazing apartment comes with \n",
       "parking, and is within walking distance to BART? The view isn't much and it does have a lot of Sec \u001b[1;36m8\u001b[0m people but is \n",
       "fairly priced for it's size and location. I'll bottom line it for ya. I need the lease and am willing to cover your\n",
       "moving expenses and knock $\u001b[1;36m300\u001b[0m off the monthly lease price. Buyer: That sounds like a great deal. I think I could \n",
       "swing $\u001b[1;36m995\u001b[0m provided I get the part in an upcoming live animation simpsons film I've auditioned for.  Seller: I \n",
       "actually know the director! I'll put in a good word for you. Seller:  Buyer: \n",
       "Output: housing\n",
       "\n",
       "Now I want you to label the following example:\n",
       "Input: Buyer:  Buyer: hi. price is quite high but i like the apt.  Seller: Your offer is too low, I can not accept \n",
       "it. I need at least $\u001b[1;36m2600\u001b[0m Seller: \n",
       "Output: \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #00ff00; text-decoration-color: #00ff00\">───────────────────────────────────────────────────────────────────────────────────────────────────────────────────</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[92m───────────────────────────────────────────────────────────────────────────────────────────────────────────────────\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# dry-run -- this tells us how much this will cost and shows an example prompt\n",
    "from autolabel import AutolabelDataset\n",
    "ds = AutolabelDataset(\"data/craigslist/test.csv\", config=config)\n",
    "agent.plan(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dd703025-54d8-4349-b0d6-736d2380e966",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-09-27 23:35:04 autolabel.labeler INFO: Task run already exists.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">There is an existing task with following details: <span style=\"color: #808000; text-decoration-color: #808000\">id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'3923857492'</span> <span style=\"color: #808000; text-decoration-color: #808000\">created_at</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">datetime</span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">.datetime</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2023</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">27</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">23</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19</span>,\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">45</span>, <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">762512</span><span style=\"font-weight: bold\">)</span> <span style=\"color: #808000; text-decoration-color: #808000\">task_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'c226c2126087831dd6926a214e453c03'</span> <span style=\"color: #808000; text-decoration-color: #808000\">dataset_id</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'7d80b97464af329f2059cce0f6b8436b'</span> \n",
       "<span style=\"color: #808000; text-decoration-color: #808000\">current_index</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span> <span style=\"color: #808000; text-decoration-color: #808000\">output_file</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'CraigslistConversationClassification_labeled.csv'</span> <span style=\"color: #808000; text-decoration-color: #808000\">status</span>=<span style=\"font-weight: bold\">&lt;</span><span style=\"color: #ff00ff; text-decoration-color: #ff00ff; font-weight: bold\">TaskStatus.ACTIVE:</span><span style=\"color: #000000; text-decoration-color: #000000\"> </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">'active'</span><span style=\"font-weight: bold\">&gt;</span> <span style=\"color: #808000; text-decoration-color: #808000\">error</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span> <span style=\"color: #808000; text-decoration-color: #808000\">metrics</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-style: italic\">None</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "There is an existing task with following details: \u001b[33mid\u001b[0m=\u001b[32m'3923857492'\u001b[0m \u001b[33mcreated_at\u001b[0m=\u001b[1;35mdatetime\u001b[0m\u001b[1;35m.datetime\u001b[0m\u001b[1m(\u001b[0m\u001b[1;36m2023\u001b[0m, \u001b[1;36m9\u001b[0m, \u001b[1;36m27\u001b[0m, \u001b[1;36m23\u001b[0m, \u001b[1;36m19\u001b[0m,\n",
       "\u001b[1;36m45\u001b[0m, \u001b[1;36m762512\u001b[0m\u001b[1m)\u001b[0m \u001b[33mtask_id\u001b[0m=\u001b[32m'c226c2126087831dd6926a214e453c03'\u001b[0m \u001b[33mdataset_id\u001b[0m=\u001b[32m'7d80b97464af329f2059cce0f6b8436b'\u001b[0m \n",
       "\u001b[33mcurrent_index\u001b[0m=\u001b[1;36m101\u001b[0m \u001b[33moutput_file\u001b[0m=\u001b[32m'CraigslistConversationClassification_labeled.csv'\u001b[0m \u001b[33mstatus\u001b[0m=\u001b[1m<\u001b[0m\u001b[1;95mTaskStatus.ACTIVE:\u001b[0m\u001b[39m \u001b[0m\n",
       "\u001b[32m'active'\u001b[0m\u001b[1m>\u001b[0m \u001b[33merror\u001b[0m=\u001b[3;35mNone\u001b[0m \u001b[33mmetrics\u001b[0m=\u001b[3;35mNone\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Evaluating the existing task<span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Evaluating the existing task\u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">classification_report:\n",
       "              precision    recall  f1-score   support\n",
       "\n",
       "        bike       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.87</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.93</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span>\n",
       "         car       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span>\n",
       " electronics       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.69</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.95</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.80</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>\n",
       "   furniture       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.96</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.82</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>\n",
       "     housing       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19</span>\n",
       "       phone       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.50</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.33</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.40</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "    accuracy                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "   macro avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.86</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.83</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.84</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "weighted avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.91</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "classification_report:\n",
       "              precision    recall  f1-score   support\n",
       "\n",
       "        bike       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.87\u001b[0m      \u001b[1;36m0.93\u001b[0m        \u001b[1;36m15\u001b[0m\n",
       "         car       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m         \u001b[1;36m9\u001b[0m\n",
       " electronics       \u001b[1;36m0.69\u001b[0m      \u001b[1;36m0.95\u001b[0m      \u001b[1;36m0.80\u001b[0m        \u001b[1;36m21\u001b[0m\n",
       "   furniture       \u001b[1;36m0.96\u001b[0m      \u001b[1;36m0.82\u001b[0m      \u001b[1;36m0.89\u001b[0m        \u001b[1;36m33\u001b[0m\n",
       "     housing       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m        \u001b[1;36m19\u001b[0m\n",
       "       phone       \u001b[1;36m0.50\u001b[0m      \u001b[1;36m0.33\u001b[0m      \u001b[1;36m0.40\u001b[0m         \u001b[1;36m3\u001b[0m\n",
       "\n",
       "    accuracy                           \u001b[1;36m0.89\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "   macro avg       \u001b[1;36m0.86\u001b[0m      \u001b[1;36m0.83\u001b[0m      \u001b[1;36m0.84\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "weighted avg       \u001b[1;36m0.91\u001b[0m      \u001b[1;36m0.89\u001b[0m      \u001b[1;36m0.89\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.89     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.89    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">101</span> examples labeled so far.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36m101\u001b[0m examples labeled so far.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Do you want to resume the task? <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">[y/n]</span>: </pre>\n"
      ],
      "text/plain": [
       "Do you want to resume the task? \u001b[1;35m[y/n]\u001b[0m: "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " y\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "31569b81f45845098fa0a4ce80c89a81",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">classification_report:\n",
       "              precision    recall  f1-score   support\n",
       "\n",
       "        bike       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.87</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.93</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">15</span>\n",
       "         car       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">9</span>\n",
       " electronics       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.69</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.95</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.80</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">21</span>\n",
       "   furniture       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.96</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.82</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">33</span>\n",
       "     housing       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1.00</span>        <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">19</span>\n",
       "       phone       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.50</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.33</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.40</span>         <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "    accuracy                           <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "   macro avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.86</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.83</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.84</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "weighted avg       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.91</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>      <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.89</span>       <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">100</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "classification_report:\n",
       "              precision    recall  f1-score   support\n",
       "\n",
       "        bike       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m0.87\u001b[0m      \u001b[1;36m0.93\u001b[0m        \u001b[1;36m15\u001b[0m\n",
       "         car       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m         \u001b[1;36m9\u001b[0m\n",
       " electronics       \u001b[1;36m0.69\u001b[0m      \u001b[1;36m0.95\u001b[0m      \u001b[1;36m0.80\u001b[0m        \u001b[1;36m21\u001b[0m\n",
       "   furniture       \u001b[1;36m0.96\u001b[0m      \u001b[1;36m0.82\u001b[0m      \u001b[1;36m0.89\u001b[0m        \u001b[1;36m33\u001b[0m\n",
       "     housing       \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m      \u001b[1;36m1.00\u001b[0m        \u001b[1;36m19\u001b[0m\n",
       "       phone       \u001b[1;36m0.50\u001b[0m      \u001b[1;36m0.33\u001b[0m      \u001b[1;36m0.40\u001b[0m         \u001b[1;36m3\u001b[0m\n",
       "\n",
       "    accuracy                           \u001b[1;36m0.89\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "   macro avg       \u001b[1;36m0.86\u001b[0m      \u001b[1;36m0.83\u001b[0m      \u001b[1;36m0.84\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "weighted avg       \u001b[1;36m0.91\u001b[0m      \u001b[1;36m0.89\u001b[0m      \u001b[1;36m0.89\u001b[0m       \u001b[1;36m100\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Actual Cost: <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Actual Cost: \u001b[1;36m0.0\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.89     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 1.0             </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.89    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m1.0            \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:15 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89386 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:15 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89386 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-13 09:27:19 openai INFO: error_code=rate_limit_exceeded error_message='Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89026 / min. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=False\n",
      "2023-08-13 09:27:19 langchain.chat_models.openai WARNING: Retrying langchain.chat_models.openai.ChatOpenAI.completion_with_retry.<locals>._completion_with_retry in 1.0 seconds as it raised RateLimitError: Rate limit reached for default-gpt-3.5-turbo in organization org-etZVkYhAIYGmLcxLmarMmAPo on tokens per min. Limit: 90000 / min. Current: 89026 / min. Contact us through our help center at help.openai.com if you continue to have issues..\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual Cost: 0.1017\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> accuracy </span>┃<span style=\"font-weight: bold\"> support </span>┃<span style=\"font-weight: bold\"> completion_rate </span>┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.7677   </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 100     </span>│<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\"> 0.99            </span>│\n",
       "└──────────┴─────────┴─────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1maccuracy\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1msupport\u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mcompletion_rate\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[1;36m \u001b[0m\u001b[1;36m0.7677  \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m100    \u001b[0m\u001b[1;36m \u001b[0m│\u001b[1;36m \u001b[0m\u001b[1;36m0.99           \u001b[0m\u001b[1;36m \u001b[0m│\n",
       "└──────────┴─────────┴─────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# now, do the actual labeling\n",
    "ds = agent.run(ds, max_items=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a0a4bc7-b62e-4ad3-96a5-9e04fe53c24b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
